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W.H.  Fleming’s  research  focussed  on  risk-sensitive  stochastic  control  prob¬ 
lems  on  an  infinite  time  horizon,  and  related  questions  in  nonlinear  i7-infinity 
control  theory.  In  risk-sensitive  control,  traditional  expected  integral  of  run¬ 
ning  cost  criteria  are  replaced  by  expected  exponential-of- integral  of  running 
cost  criteria.  On  an  infinite  time  horizon,  the  goal  is  to  optimize  the  long  term 
growth  rate  of  such  risk  sensitive  criteria.  By  introducing  a  small  parameter 
which  corresponds  to  a  disturbance  error  intensity,  dynamic  games  which  arise 
in  iJ- infinity  control  are  obtained  in  the  zero  disturbance  error  limit.  This  tech¬ 
nique  provides  a  link  between  stochastic  and  deterministic  approaches  to  the 
disturbance  attenuation  problem  for  control  systems. 

Publication  1  is  concerned  with  infinite  horizon,  risk  sensitive  control  of 
finite  state  machines.  In  this  case,  states,  controls  and  disturbances  all  belong 
to  finite  sets.  The  finite  state  machine  setting  avoids  purely  technical  issues 
which  may  obscure  the  main  ideas  in  the  development.  Part  I  of  Publication  1 
concerns  state  feedback  control,  and  Part  II  concerns  output  feedback  (control 
with  partial  state  information).  For  the  output  feedback  case,  the  finite  state 
machine  i!f-infinity  control  problem  was  approached  directly,  without  obtaining 
it  as  a  zero  disturbance  error  limit. 

Publication  2  is  concerned  with  a  production  planning  system,  in  which 
machine  breakdowns  and  repairs  occur  on  a  much  faster  time  scale  than  the 
underlying  planning  scale.  The  zero  disturbance  error  limit  corresponds  in  this 
problem  to  a  kind  of  averaging  out  of  the  fast  time  scale  effects. 

Reference  3  is  a  survey  of  recent  results  and  open  problems  in  risk  sensitive 
control. 

Risk  sensitive  control  theory  is  related  to  problems  of  large  deviations  for 
Markov  stochastic  processes,  and  of  eigenvalues  and  eigenfunctions  of  associated 
linear  partial  differential  operators.  Reference  4  is  concerned  with  such  prob¬ 
lems,  in  cases  when  the  Markov  process  is  an  ergodic  diffusion  which  is  nearly 
deterministic. 

The  following  describes  the  recent  work  of  Kushner.  New  and  very  practical 
convergence  results  for  closed  loop  adaptive  noise  cancellation  were  obtained. 
Given  the  mean  limit  ODE  for  the  stochastic  approximation  defining  the  adap¬ 
tive  algorithm  for  a  closed  loop  ANC,  we  characterize  the  limit  points.  Under 
appropriate  conditions,  it  is  shown  that  as  the  dimension  of  the  weight  vector 
increases,  the  sequence  of  corresponding  limit  points  converges  in  the  sense  of 
I2  to  the  infinite  dimensional  optimal  weight  vector.  Also,  the  limit  point  of 
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the  algorithm  is  nearly  optimal  if  the  dimension  of  the  weight  vector  is  large 
enough.  The  gradient  of  the  mean  square  error  with  respect  to  the  weight  vec¬ 
tor,  evaluated  at  the  limit,  goes  to  zero  in  /i  and  I2  as  the  dimension  increases, 
as  does  the  gradient  with  respect  to  the  coefficients  in  the  transfer  function 
connecting  the  reference  noise  signal  with  the  error  output.  Thus  the  algorithm 
is  “nearly”  a  gradient  descent  algorithm  and  is  indeed  error  reducing  for  large 
enough  dimension.  Under  broad  conditions,  iterate  averaging  can  be  used  to 
get  a  nearly  optimal  rate  of  convergence. 

Much  work  was  done  on  problems  in  modern  high  speed  telecommunications. 
A  large  part  of  the  aim  was  the  demonstration  of  the  usefulness  of  numerical 
methods  of  stochastic  control  theory  for  the  design,  analysis  and  control  of  mul¬ 
tiplexing  type  systems  and  networks,  as  well  as  of  the  larger  ATM  type  systems 
of  which  they  form  basic  components.  By  exploiting  the  large  size  of  the  sys¬ 
tem  (large  number  of  users),  the  systems  can  be  efficiently  approximated  by 
diffusion  type  processes,  whether  there  is  a  control  term  or  not,  and  for  many 
types  of  control  mechanisms.  The  basic  controls  are  of  the  “low  priority  cell 
deletion”  type,  and  various  extensions.  They  might  be  state  dependent,  and 
we  can  obtain  optimal  controls  for  cost  functions  that  weigh  buffer  overflow, 
controller  cell  deletion  loss  as  well  as  queue  length.  The  limit  equations  are  an 
effective  aggregation  of  the  original  system.  They  can  be  used  for  numerical  as 
well  as  analytical  work,  and  the  numerical  methods  are  based  on  them.  Many 
forms  of  the  problem  have  the  same  limit  equations.  The  structure  of  the  nu¬ 
merical  approximations  is  close  to  that  of  the  original  problem,,  but  is  much 
simpler.  It  is  shown  that  there  are  substantial  savings  in  losses  with  the  use 
of  optimal  control  techniques.  The  optimal  controls  often  have  a  surprisingly 
simple  structure.  The  numerical  methods  can  be  used  to  get  controls  which 
appropriately  balance  the  losses  at  the  control  with  those  due  to  buffer  over¬ 
flow.  They  can  be  used  to  get  controls  which  minimize  losses  at  the  controller 
subject  to  constraints  on  buffer  overflow.  They  can  be  used  to  explore  various 
approximations,  systems  aggregations,  the  interaction  of  multiple  source  classes 
of  different  priorities.  These  are  typical  of  many  possibilities.  The  approach  is 
a  powerful  useful  tool  for  getting  both  qualitative  and  quantitative  information 
on  problems  which  would  be  hard  to  do  otherwise. 

Perhaps  the  major  achievement  was  the  completion  of  a  book  on  Stochas¬ 
tic  Approximation  or  recursive  stochastic  algorithms,  for  both  constrained  and 
unconstrained  problems,  no  doubt  the  best  and  most  comprehensive  to  date. 
The  general  motivation  arises  from  the  new  challenges  in  applications  that  have 
arisen  in  recent  years.  There  is  a  thorough  treatment  of  both  probability  one 
and  weak  convergence  methods  for  very  general  noise  models.  The  conver¬ 
gence  proofs  are  built  around  the  powerful  ODE  (ordinary  differential  equations 
method),  which  characterizes  the  asymptotic  points  of  the  algorithm  in  terms 
of  the  asymptotic  behavior  of  a  “mean  limit  ODE”  or  analogous  dynamical 
system.  The  method  is  particularly  convenient  for  dealing  with  complex  noise 
and  dynamics,  but  also  greatly  simplifies  the  treatment  of  the  more  classical 
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cases.  There  is  a  thorough  treatment  of  rate  of  convergence,  iterate  averaging, 
high  dimensional  problems,  ergodic  cost  problems,  stability  methods  for  corre¬ 
lated  noise,  and  decentralized  and  asynchronous  algorithms.  The  basic  ideas 
concerning  the  application  of  large  deviations  methods  are  discussed.  The  use 
of  perturbed  test  function  or  perturbed  state  methods  for  complex  correlated 
noise  processes  is  explained.  These  methods  are  applicable  generally  to  ap¬ 
proximation  and  limit  problems.  The  development  proceeds  from  the  simpler 
problems  to  the  more  complex,  allowing  the  underlying  ideas  to  be  motivated 
and  understood  without  excessive  encumbrance.  Many  motivating  examples 
are  described,  and  convergence  proofs  for  a  wide  range  of  problems  (signal  pro¬ 
cessing,  learning  theory,  ergodic  cost  problems,  noise  cancellation,  neural  net 
training,  etc)  are  given  in  order  to  illustrate  the  methods. 
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